<?xml version="1.0" encoding="utf-8" standalone="no"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
  "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">

<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" xmlns:xml="http://www.w3.org/XML/1998/namespace">
<head>
  <meta content="width=device-width, initial-scale=1.0" name="viewport" />
  <meta content="$Id: yesod.html 1361 2012-05-04 16:08:53Z amy $" name="provenance" />
  <link href="../Styles/bootstrap.css" rel="stylesheet" type="text/css" />
  <link href="../Styles/bootstrap-responsive.css" rel="stylesheet" type="text/css" />
  <link href="../Styles/aosa.css" rel="stylesheet" type="text/css" />

  <title>The Architecture of Open Source Applications (Volume 2): Yesod</title>
</head>

<body>
  <div class="container row-fluid hero-unit">
    <h1 id="heading_id_2">Yesod</h1>

    <p><a href="../Text/intro2.html#snoyman-michael">Michael Snoyman</a></p>
  </div>

  <p>Yesod is a web framework written in the Haskell programming language. While many popular web frameworks exploit the dynamic nature of their host languages, Yesod exploits the static nature of Haskell to produce safer, faster code.</p>

  <p>Development began about two years ago and has been going strong ever since. Yesod cut its teeth on real life projects, with all of its initial features born out of an actual, real-life need. At first, development was almost entirely a one-man show. After about a year of development the community efforts kicked in, and Yesod has since blossomed into a thriving open source project.</p>

  <p>During the embryonic phase, when Yesod was incredibly ephemeral and ill-defined, it would have been counter-productive to try and get a team to work on it. By the time it stabilized enough to be useful to others, it was the right time to find out the downsides to some of the decisions that had been made. Since then, we have made major changes to the user-facing API to make it more useful, and are quickly solidifying a 1.0 release.</p>

  <p>The question you may ask is: Why another web framework? Let's instead redirect to a different question: Why use Haskell? It seems that most of the world is happy with one of two styles of language:</p>

  <ul>
    <li>Statically typed languages, like Java, C# and C++. These languages provide speed and type safety, but are more cumbersome to program with.</li>

    <li>Dynamically typed languages, like Ruby and Python. These languages greatly increase productivity (at least in the short run), but run slowly and have very little support from the compiler to ensure correctness. (The solution to this last point is unit testing. We'll get to that later.)</li>
  </ul>

  <p>This is a false dichotomy. There's no reason why statically typed languages need to be so clumsy. Haskell is able to capture a huge amount of the expressivity of Ruby and Python, while remaining a strongly typed language. In fact, Haskell's type system catches many more bugs than Java and its ilk. Null pointer exceptions are completely eliminated; immutable data structures simplify reasoning about your code and simplify parallel and concurrent programming.</p>

  <p>So why Haskell? It is an efficient, developer-friendly language which provides many compile-time checks of program correctness.</p>

  <p>The goal of Yesod is to extend Haskell's strengths into web development. Yesod strives to make your code as concise as possible. As much as possible, every line of your code is checked for correctness at compile time. Instead of requiring large libraries of unit tests to test basic properties, the compiler does it all for you. Under the surface, Yesod uses as many advanced performance techniques as we can muster to make your high-level code fly.</p>

  <h2 id="heading_id_3">22.1. Compared to Other Frameworks</h2>

  <p>In general terms, Yesod is more similar to than different than the leading frameworks such as Rails and Django. It generally follows the Model-View-Controller (MVC) paradigm, has a templating system that separates view from logic, provides an Object-Relational Mapping (ORM) system, and has a front controller approach to routing.</p>

  <p>The devil is in the details. Yesod strives to push as much error catching to the compile phase instead of runtime, and to automatically catch both bugs and security flaws through the type system. While Yesod tries to maintain a user-friendly, high-level API, it uses a number of newer techniques from the functional programming world to achieve high performance, and is not afraid to expose these internals to developers.</p>

  <p>The main architectural challenge in Yesod is balancing these two seemingly conflicting goals. For example, there is nothing revolutionary about Yesod's <a href="http://www.yesodweb.com/blog/2012/01/aosa-chapter#file1414-routes">approach to routing</a> (called <em>type-safe URLs</em>). Historically, implementing such a solution was a tedious, error-prone process. Yesod's innovation is to use Template Haskell (a form of code generation) to automate the boilerplate required to bootstrap the process. Similarly, type-safe HTML has been around for a long while; Yesod tries to keep the developer-friendly aspect of common template languages while keeping the power of type safety.</p>

  <h2 id="heading_id_4">22.2. Web Application Interface</h2>

  <p>A web application needs some way to communicate with a server. One possible approach is to bake the server directly into the framework, but doing so necessarily limits your options for deployment and leads to poor interfaces. Many languages have created standard interfaces to address this issue: Python has WSGI and Ruby has Rack. In Haskell, we have WAI: Web Application Interface.</p>

  <p>WAI is not intended to be a high-level interface. It has two specific goals: generality and performance. By staying general, WAI has been able to support backends for everything from standalone servers to old school CGI and even works directly with Webkit to produce faux desktop applications. The performance side will introduce us to a number of the cool features of Haskell.</p>

  <div class="figure" id="fig.yesod.overall">
    <img alt="" src="../Images/overview.png" />

    <p>Figure 22.1: Overall structure of a Yesod application</p>
  </div>

  <h3 id="heading_id_5">Datatypes</h3>

  <p>One of the biggest advantages of Haskell—and one of the things we make the most use of in Yesod—is strong static typing. Before we begin to write the code for how to solve something, we need to think about what the data will look like. WAI is a perfect example of this paradigm. The core concept we want to express is that of an application. An application's most basic expression is a function that takes a request and returns a response. In Haskell lingo:</p>
  <pre>
type Application = Request -&gt; Response
</pre>

  <p>This just raises the question: what do <code>Request</code> and <code>Response</code> look like? A request has a number of pieces of information, but the most basic are the requested path, query string, request headers, and request body. And a response has just three components: a status code, response headers and response body.</p>

  <p>How do we represent something like a query string? Haskell keeps a strict separation between binary and textual data. The former is represented by <code>ByteString</code>, the latter by <code>Text</code>. Both are highly optimized datatypes that provide a high-level, safe API. In the case of a query string we store the raw bytes transferred over the wire as a <code>ByteString</code> and the parsed, decoded values as <code>Text</code>.</p>

  <h3 id="heading_id_6">Streaming</h3>

  <p>A <code>ByteString</code> represents a single memory buffer. If we were to naively use a plain <code>ByteString</code> for holding the entire request or response bodies, our applications could never scale to large requests or responses. Instead, we use a technique called enumerators, very similar in concept to generators in Python. Our <code>Application</code> becomes a consumer of a stream of <code>ByteString</code>s representing the incoming request body, and a producer of a separate stream for the response.</p>

  <p>We now need to slightly revise our definition of an <code>Application</code>. An <code>Application</code> will take a <code>Request</code> value, containing headers, query string, etc., and will consume a stream of <code>ByteString</code>s, producing a <code>Response</code>. So the revised definition of an <code>Application</code> is:</p>
  <pre>
type Application = Request -&gt; Iteratee ByteString IO Response
</pre>

  <p>The <code>IO</code> simply explains what types of side effects an application can perform. In the case of <code>IO</code>, it can perform any kind of interaction with the outside world, an obvious necessity for the vast majority of web applications.</p>

  <h3 id="heading_id_7">Builder</h3>

  <p>The trick in our arsenal is how we produce our response buffers. We have two competing desires here: minimizing system calls, and minimizing buffer copies. On the one hand, we want to minimize system calls for sending data over the socket. To do this we need to store outgoing data in a buffer. However, if we make this buffer too large, we will exhaust our memory and slow down the application's response time. On the other hand, we want to minimize the number of times data is copied between buffers, preferably copying just once from the source to destination buffer.</p>

  <p>Haskell's solution is the <em>builder</em>. A builder is an instruction for how to fill a memory buffer, such as: place the five bytes "hello" in the next open position. Instead of passing a stream of memory buffers to the server, a WAI application passes a stream of these instructions. The server takes the stream and uses it to fill up optimally sized memory buffers. As each buffer is filled, the server makes a system call to send the data over over the wire and then starts filling up the next buffer.</p>

  <p>(The optimal size for a buffer will depend on many factors such as cache size. The underlying blaze-builder library underwent significant performance testing to determine the best trade-off.)</p>

  <p>In theory, this kind of optimization could be performed in the application itself. However, by encoding this approach in the interface, we are able to simply prepend the response headers to the response body. The result is that, for small to medium-sized responses, the entire response can be sent with a single system call and memory is copied only once.</p>

  <h3 id="heading_id_8">Handlers</h3>

  <p>Now that we have an application, we need some way to run it. In WAI parlance, this is a <em>handler</em>. WAI has some basic, standard handlers, such as the standalone server Warp (discussed below), FastCGI, SCGI and CGI. This spectrum allows WAI applications to be run on anything from dedicated servers to shared hosting. But in addition to these, WAI has some more interesting backends:</p>

  <ul>
    <li><strong>Webkit:</strong> This backend embeds a Warp server and calls out to QtWebkit. By launching a server, then launching a new standalone browser window, we have faux desktop applications.</li>

    <li><strong>Launch:</strong> This is a slight variant on Webkit. Having to deploy the Qt and Webkit libraries can be a bit burdensome, so instead we just launch the user's default browser.</li>

    <li><strong>Test:</strong> Even testing counts as a handler. After all, testing is simply the act of running an application and inspecting the responses.</li>
  </ul>

  <p>Most developers will likely use Warp. It is lightweight enough to be used for testing. It requires no config files, no folder hierarchy and no long-running, administrator-owned process. It's a simple library that gets compiled into your application or run via the Haskell interpreter. Warp is an incredibly fast server, with protection from all kinds of attack vectors, such as Slowloris and infinite headers. Warp can be the only web server you need, though it is also quite happy to sit behind a reverse HTTP proxy.</p>

  <p>The PONG benchmark measures the requests per second of various servers for the 4-byte response body "PONG". In the graph shown in <a href="#fig.yesod.warp">Figure 22.2</a>, Yesod is measured as a framework on top of Warp. As can be seen, the Haskell servers (Warp, Happstack and Snap) lead the pack.</p>

  <div class="figure" id="fig.yesod.warp">
    <img alt="" src="../Images/extra-large.png" />

    <p>Figure 22.2: Warp PONG benchmark</p>
  </div>

  <p>Most of the reasons for Warp's speed have already been spelled out in the overall description of WAI: enumerators, builders and packed datatypes. The last piece in the puzzle is from the Glasgow Haskell Compiler's (GHC's) multithreaded runtime. GHC, Haskell's flagship compiler, has light-weight green threads. Unlike system threads, it is possible to spin up thousands of these without serious performance hits. Therefore, in Warp each connection is handled by its own green thread.</p>

  <p>The next trick is asynchronous I/O. Any web server hoping to scale to tens of thousands of requests per second will need some type of asynchronous communication. In most languages, this involves complicated programming involving callbacks. GHC lets us cheat: we program as if we're using a synchronous API, and GHC automatically switches between different green threads waiting for activity.</p>

  <p>Under the surface, GHC uses whatever system is provided by the host operating system, such as <code>kqueue</code>, <code>epoll</code> and <code>select</code>. This gives us all the performance of an event-based I/O system, without worrying about cross-platform issues or writing in a callback-oriented way.</p>

  <h3 id="heading_id_9">Middleware</h3>

  <p>In between handlers and applications, we have <code>middleware</code>. Technically, middleware is an <em>application transformer</em>: it takes one <code>Application</code>, and returns a new one. This is defined as:</p>
  <pre>
type Middleware = Application -&gt; Application
</pre>

  <p>The best way to understand the purpose of middleware is to look at some common examples:</p>

  <ul>
    <li><code>gzip</code> automatically compresses the response from an application.</li>

    <li><code>jsonp</code> automatically converts JSON responses to JSON-P responses when the client provided a callback parameter.</li>

    <li><code>autohead</code> will generate appropriate HEAD responses based on the GET response of an application.</li>

    <li><code>debug</code> will print debug information to the console or a log on each request.</li>
  </ul>

  <p>The idea here is to factor out common code from applications and let it be shared easily. Note that, based on the definition of middleware, we can easily stack these things up. The general workflow of middleware is:</p>

  <ol>
    <li>Take the request value and apply some modifications.</li>

    <li>Pass the modified request to the application and receive a response.</li>

    <li>Modify the response and return it to the handler.</li>
  </ol>

  <p>In the case of stacked middleware, instead of passing to the application or handler, the in-between middleware will actually be passing to the inner and outer middleware, respectively.</p>

  <h3 id="heading_id_10">Wai-test</h3>

  <p>No amount of static typing will obviate the need for testing. We all know that automated testing is a necessity for any serious applications. <code>wai-test</code> is the recommended approach to testing a WAI application. Since requests and responses are simple datatypes, it is easy to mock up a fake request, pass it to an application, and test properties about the response. <code>wai-test</code> simply provides some convenience functions for testing common properties like the presence of a header or a status code.</p>

  <h2 id="heading_id_11">22.3. Templates</h2>

  <p>In the typical Model-View-Controller (MVC) paradigm, one of the goals is to separate logic from the view. Part of this separation is achieved through the use of a template language. However, there are many different ways to approach this issue. At one end of the spectrum, for example, PHP/ASP/JSP will allow you to embed any arbitrary code within your template. At the other end, you have systems like StringTemplate and QuickSilver, which are passed some arguments and have no other way of interacting with the rest of the program.</p>

  <p>Each system has its pros and cons. Having a more powerful template system can be a huge convenience. Need to show the contents of a database table? No problem, pull it in with the template. However, such an approach can quickly lead to convoluted code, interspersing database cursor updates with HTML generation. This can be commonly seen in a poorly written ASP project.</p>

  <p>While weak template systems make for simple code, they also tend towards a lot of redundant work. You will often need to not only keep your original values in datatypes, but also create dictionaries of values to pass to the template. Maintaining such code is not easy, and usually there is no way for a compiler to help you out.</p>

  <p>Yesod's family of template languages, the Shakespearean languages, strive for a middle ground. By leveraging Haskell's standard referential transparency, we can be assured that our templates produce no side effects. However, they still have full access to all the variables and functions available in your Haskell code. Also, since they are fully checked for both well-formedness, variable resolution and type safety at compile time, typos are much less likely to have you searching through your code trying to pin down a bug.</p>

  <div class="box">
    <p class="boxtitle">Why the Name Shakespeare?</p>

    <p>The HTML language, Hamlet, was the first language written, and originally based its syntax on Haml. Since it was at the time a "reduced" Haml, Hamlet seemed appropriate. As we added CSS and Javascript options, we decided to keep the naming theme with Cassius and Julius. At this point, Hamlet looks nothing like Haml, but the name stuck anyway.</p>
  </div>

  <h3 id="heading_id_12">Types</h3>

  <p>One of the overarching themes in Yesod is proper use of types to make developers' lives easier. In Yesod templates, we have two main examples:</p>

  <ol>
    <li>All content embedded into a Hamlet template must have a type of <code>Html</code>. As we'll see later, this forces us to properly escape dangerous HTML when necessary, while avoiding accidental double-escaping as well.</li>

    <li>Instead of concatenating URLs directly in our template, we have datatypes—known as type-safe URLs—which represent the routes in our application.</li>
  </ol>

  <p>As a real-life example, suppose that a user submits his/her name to an application via a form. This data would be represented with the <code>Text</code> datatype. Now we would like to display this variable, called <code>name</code>, in a page. The type system—at compile time—prevents it from being simply stuck into a Hamlet template, since it's not of type <code>Html</code>. Instead we must convert it somehow. For this, there are two conversion functions:</p>

  <ol>
    <li><code>toHtml</code> will automatically escape any entities. So if a user submits the string <code>&lt;script src="http://example.com/evil.js"&gt;&lt;/script&gt;</code>, the less-than signs will automatically be converted to <code>&amp;lt;</code>.</li>

    <li><code>preEscapedText</code>, on the other hand, will leave the content precisely as it is now.</li>
  </ol>

  <p>So in the case of untrusted input from a possibly nefarious user, <code>toHtml</code> would be our recommended approach. On the other hand, let us say we have some static HTML stored on our server that we would like to insert into some pages verbatim. In that case, we could load it into a <code>Text</code> value and then apply <code>preEscapedText</code>, thereby avoiding any double-escaping.</p>

  <p>By default, Hamlet will use the <code>toHtml</code> function on any content you try to interpolate. Therefore, you only need to explicitly perform a conversion if you want to avoid escaping. This follows the dictum of erring on the side of caution.</p>
  <pre>
name &lt;- runInputPost $ ireq textField "name"
snippet &lt;- readFile "mysnippet.html"
return [hamlet|
    &lt;p&gt;Welcome #{name}, you are on my site!
    &lt;div .copyright&gt;#{preEscapedText snippet}
|]
</pre>

  <p>The first step in type-safe URLs is creating a datatype that represents all the routes in your site. Let us say you have a site for displaying Fibonacci numbers. The site will have a separate page for each number in the sequence, plus the homepage. This could be modeled with the Haskell datatype:</p>
  <pre>
data FibRoute = Home | Fib Int
</pre>

  <p>We could then create a page like so:</p>
  <pre>
&lt;p&gt;You are currently viewing number #{show index} in the sequence. Its value is #{fib index}.
&lt;p&gt;
    &lt;a href=@{Fib (index + 1)}&gt;Next number
&lt;p&gt;
    &lt;a href=@{Home}&gt;Homepage
</pre>

  <p>Then all we need is some function to convert a type-safe URL into a string representation. In our case, that could look something like this:</p>
  <pre>
render :: FibRoute -&gt; Text
render Home = "/home"
render (Fib i) = "/fib/" ++ show i
</pre>

  <p>Fortunately, all of the boilerplate of defining and rendering type-safe URL datatypes is handled for the developer automatically by Yesod. We will cover that in more depth later.</p>

  <h3 id="heading_id_13">The Other Languages</h3>

  <p>In addition to Hamlet, there are three other languages: Julius, Cassius and Lucius. Julius is used for Javascript; however, it's a simple pass-through language, just allowing for interpolation. In other words, barring accidental use of the interpolation syntax, any piece of Javascript could be dropped into Julius and be valid. For example, to test the performance of Julius, jQuery was run through the language without an issue.</p>

  <p>The other two languages are alternate CSS syntaxes. Those familiar with the difference between Sass and Less will recognize this immediately: Cassius is whitespace delimited, while Lucius uses braces. Lucius is in fact a superset of CSS, meaning all valid CSS files are valid Lucius files. In addition to allowing text interpolation, there are some helper datatypes provided to model unit sizes and colors. Also, type-safe URLs work in these languages, making it convenient for specifying background images.</p>

  <p>Aside from the type safety and compile-time checks mentioned above, having specialized languages for CSS and Javascript give us a few other advantages:</p>

  <ul>
    <li>For production, all the CSS and Javascript is compiled into the final executable, increasing performance (by avoiding file I/O) and simplifying deployment.</li>

    <li>By being based around the efficient builder construct described earlier, the templates can be rendered very quickly.</li>

    <li>There is built-in support for automatically including these in final webpages. We will get into this in more detail when describing widgets below.</li>
  </ul>

  <h2 id="heading_id_14">22.4. Persistent</h2>

  <p>Most web applications will want to store information in a database. Traditionally, this has meant some kind of SQL database. In that regard, Yesod continues a long tradition, with PostgreSQL as our most commonly used backend. But as we have been seeing in recent years, SQL isn't always the answer to the persistence question. Therefore, Yesod was designed to work well with NoSQL databases as well, and ships with a MongoDB backend as a first-class citizen.</p>

  <p>The result of this design decision is Persistent, Yesod's preferred storage option. There are really two guiding lights for Persistent: make it as back-end-agnostic as possible, and let user code be completely type-checked.</p>

  <p>At the same time, we fully recognize that it is impossible to completely shield the user from all details of the backend. Therefore, we provide two types of escape routes:</p>

  <ul>
    <li>Back-end-specific functionality as necessary. For example, Persistent provides features for SQL joins and MongoDB lists and hashes. Proper portability warnings will apply, but if you want this functionality, it's there.</li>

    <li>Easy access to performing raw queries. We don't believe it's possible for any abstraction to cover every use case of the underlying library. If you just have to write a 5-table, correlated subquery in SQL, go right ahead.</li>
  </ul>

  <h3 id="heading_id_15">Terminology</h3>

  <p>The most primitive datatype in Persistent is the <code>PersistValue</code>. This represents any raw data that can appear within the database, such as a number, a date, or a string. Of course, sometimes you'll have some more user-friendly datatypes you want to store, like HTML. For that, we have the <code>PersistField</code> class. Internally, a <code>PersistField</code> expresses itself to the database in terms of a <code>PersistValue</code>.</p>

  <p>All of this is very nice, but we will want to combine different fields together into a larger picture. For this, we have a <code>PersistEntity</code>, which is basically a collection of <code>PersistField</code>s. And finally, we have a <code>PersistBackend</code> that describes how to create, read, update and delete these entities.</p>

  <p>As a practical example, consider storing a person in a database. We want to store the person's name, birthday, and a profile image (a PNG file). We create a new entity <code>Person</code> with three fields: a <code>Text</code>, a <code>Day</code> and a <code>PNG</code>. Each of those gets stored in the database using a different <code>PersistValue</code> constructor: <code>PersistText</code>, <code>PersistDay</code> and <code>PersistByteString</code>, respectively.</p>

  <p>There is nothing surprising about the first two mappings, but the last one is interesting. There is no specific constructor for storing PNG content in a database, so instead we use a more generic type (a <code>ByteString</code>, which is just a sequence of bytes). We could use the same mechanism to store other types of arbitrary data.</p>

  <p>(The commonly held best practice for storing images is to keep the data on the filesystem and just keep a path to the image in the database. We do not advocate against using that approach, but are rather using database-stored images as an illustrative example.)</p>

  <p>How is all this represented in the database? Consider SQL as an example: the <code>Person</code> entity becomes a table with three columns (name, birthday, and picture). Each field is stored as a different SQL type: <code>Text</code> becomes a <code>VARCHAR</code>, <code>Day</code> becomes a <code>Date</code> and <code>PNG</code> becomes a <code>BLOB</code> (or <code>BYTEA</code>).</p>

  <p>The story for MongoDB is very similar. <code>Person</code> becomes its own <em>document</em>, and its three fields each become a MongoDB <em>field</em>. There is no need for datatypes or creation of a schema in MongoDB.</p>

  <table class="table table-condensed">
    <tr>
      <td>Persistent</td>

      <td>SQL</td>

      <td>MongoDB</td>
    </tr>

    <tr>
      <td>PersistEntity</td>

      <td>Table</td>

      <td>Document</td>
    </tr>

    <tr>
      <td>PersistField</td>

      <td>Column</td>

      <td>Field</td>
    </tr>

    <tr>
      <td>PersistValue</td>

      <td>Column type</td>

      <td>N/A</td>
    </tr>
  </table>

  <h3 id="heading_id_16">Type Safety</h3>

  <p>Persistent handles all of the data marshaling concerns behind the scenes. As a user of Persistent, you get to completely ignore the fact that a <code>Text</code> becomes a <code>VARCHAR</code>. You are able to simply declare your datatypes and use them.</p>

  <p>Every interaction with Persistent is strongly typed. This prevents you from accidentally putting a number in the date fields; the compiler will not accept it. Entire classes of subtle bugs simply disappear at this point.</p>

  <p>Nowhere is the power of strong typing more pronounced than in refactoring. Let's say you have been storing users' ages in the database, and you realize that you really wanted to store birthdays instead. You are able to make a single line change to your entities declaration file, hit compile, and automatically find every single line of code that needs to be updated.</p>

  <p>In most dynamically-typed languages, and their web frameworks, the recommended approach to solving this issue is writing unit tests. If you have full test coverage, then running your tests will immediately reveal what code needs to be updated. This is all well and good, but it is a weaker solution than true types:</p>

  <ul>
    <li>It is all predicated on having full test coverage. This takes extra time, and worse, is boilerplate code that the compiler should be able to do for you.</li>

    <li>You might be a perfect developer who never forgets to write a test, but can you say the same for every person who will touch your codebase?</li>

    <li>Even 100% test coverage doesn't guarantee that you really have tested every case. All it's done is proven you've tested every line of code.</li>
  </ul>

  <h3 id="heading_id_17">Cross-Database Syntax</h3>

  <p>Creating an SQL schema that works for multiple SQL engines can be tricky enough. How do you create a schema that will also work with a non-SQL database like MongoDB?</p>

  <p>Persistent allows you to define your entities in a high-level syntax, and will automatically create the SQL schema for you. In the case of MongoDB, we currently use a schema-less approach. This also allows Persistent to ensure that your Haskell datatypes match perfectly with the database's definitions.</p>

  <p>Additionally, having all this information gives Persistent the ability to perform more advanced functions, such as migrations, for you automatically.</p>

  <h3 id="heading_id_18">Migrations</h3>

  <p>Persistent not only creates schema files as necessary, but will also automatically apply database migrations if possible. Database modification is one of the less-developed pieces of the SQL standard, and thus each engine has a different take on the process. As such, each Persistent backend defines its own set of migration rules. In PostgreSQL, which has a rich set of <code>ALTER TABLE</code> rules, we use those extensively. Since SQLite lacks much of that functionality, we are reduced to creating temporary tables and copying rows. MongoDB's schema-less approach means no migration support is required.</p>

  <p>This feature is purposely limited to prevent any kind of data loss. It will not remove any columns automatically; instead, it will give you an error message, telling you the unsafe operations that are necessary in order to continue. You will then have the option of either manually running the SQL it provides you, or changing your data model to avoid the dangerous behavior.</p>

  <h3 id="heading_id_19">Relations</h3>

  <p>Persistent is non-relational in nature, meaning it has no requirement for backends to support relations. However, in many use cases, we may want to use relations. In those cases, developers will have full access to them.</p>

  <p>Assume we want to now store a list of skills with each user. If we were writing a MongoDB-specific app, we could go ahead and just store that list as a new field in the original <code>Person</code> entity. But that approach would not work in SQL. In SQL, we call this kind of relationship a one-to-many relationship.</p>

  <p>The idea is to store a reference to the "one" entity (person) with each "many" entity (skill). Then if we want to find all the skills a person has, we simply find all skills that reference that person. For this reference, every entity has an ID. And as you might expect by now, these IDs are completely type-safe. The datatype for a Person ID is <code>PersonId</code>. So to add our new skill, we would just add the following to our entity definition:</p>
  <pre>
Skill
    person PersonId
    name Text
    description Text
    UniqueSkill person name
</pre>

  <p>This ID datatype concept comes up throughout Persistent and Yesod. You can dispatch based on an ID. In such a case, Yesod will automatically marshal the textual representation of the ID to the internal one, catching any parse errors along the way. These IDs are used for lookup and deletion with the <code>get</code> and <code>delete</code> functions, and are returned by the insertion and query functions <code>insert</code> and <code>selectList</code>.</p>

  <h2 id="heading_id_20">22.5. Yesod</h2>

  <p>If we are looking at the typical Model-View-Controller (MVC) paradigm, Persistent is the model and Shakespeare is the view. This would leave Yesod as the controller.</p>

  <p>The most basic feature of Yesod is routing. It features a declarative syntax and type-safe dispatch. Layered on top of this, Yesod provides many other features: streaming content generation, widgets, i18n, static files, forms and authentication. But the core feature added by Yesod is really routing.</p>

  <p>This layered approach makes it simpler for users to swap different components of the system. Some people are not interested in using Persistent. For them, nothing in the core system even mentions Persistent. Likewise, while they are commonly used features, not everyone needs authentication or static file serving.</p>

  <p>On the other hand, many users <em>will</em> want to integrate all of these features. And doing so, while enabling all the optimizations available in Yesod, is not always straightforward. To simplify the process, Yesod also provides a scaffolding tool that sets up a basic site with the most commonly used features.</p>

  <h3 id="heading_id_21">Routes</h3>

  <p>Given that routing is really the main function of Yesod, let's start there. The routing syntax is very simple: a <em>resource pattern</em>, a name, and request methods. For example, a simple blog site might look like:</p>
  <pre>
/ HomepageR GET
/add-entry AddEntryR GET POST
/entry/#EntryId EntryR GET
</pre>

  <p>The first line defines the homepage. This says "I respond to the root path of the domain, I'm called HomepageR, and I answer GET requests." (The trailing "R" on the resource names is simply a convention, it doesn't hold any special meaning besides giving a cue to the developer that something is a route.)</p>

  <p>The second line defines the add-entry page. This time, we answer both GET and POST requests. You might be wondering why Yesod, as opposed to most frameworks, requires you to explicitly state your request methods. The reason is that Yesod tries to adhere to RESTful principles as much as possible, and GET and POST requests really have very different meanings. Not only do you state these two methods separately, but later you will define their handler functions separately. (This is actually an optional feature in Yesod. If you want, you can leave off the list of methods and your handler function will deal with all methods.)</p>

  <p>The third line is a bit more interesting. After the second slash we have <code>#EntryId</code>. This defines a parameter of type <code>EntryId</code>. We already alluded to this feature in the Persistent section: Yesod will now automatically marshal the path component into the relevant ID value. Assuming an SQL backend (Mongo is addressed later), if a user requests <code>/entry/5</code>, the handler function will get called with an argument <code>EntryId 5</code>. But if the user requests <code>/entry/some-blog-post</code>, Yesod will return a 404.</p>

  <p>This is obviously possible in most other web frameworks as well. The approach taken by Django, for instance, would use a regular expression for matching the routes, e.g. <code>r"/entry/(\d+)"</code>. The Yesod approach, however, provides some advantages:</p>

  <ul>
    <li>Typing "EntryId" is much more semantic/developer-friendly than a regular expression.</li>

    <li>Regular expressions cannot express everything (or at least, can't do so succinctly). We can use <code>/calendar/#Day</code> in Yesod; do you want to type a regex to match dates in your routes?</li>

    <li>Yesod also automatically marshals the data for us. In our calendar case, our handler function would receive a <code>Day</code> value. In the Django equivalent, the function would receive a piece of text which it would then have to marshal itself. This is tedious, repetitive and inefficient.</li>

    <li>So far we've assumed that a database ID is just a string of digits. But what if it's more complicated? MongoDB uses GUIDs, for example. In Yesod, your <code>#EntryId</code> will still work, and the type system will instruct Yesod how to parse the route. In a regex system, you would have to go through all of your routes and change the <code>\d+</code> to whatever monstrosity of regex is needed to match GUIDs.</li>
  </ul>

  <h4 id="heading_id_22">Type-Safe URLs</h4>

  <p>This approach to routing gives birth to one of Yesod's most powerful features: type-safe URLs. Instead of just splicing together pieces of text to refer to a route, every route in your application can be represented by a Haskell value. This immediately eliminates a large number of 404 Not Found errors: it is simply not possible to produce an invalid URL. (It is still possible to produce a URL that would lead to a 404 error, such as by referring to a blog post that does not exist. However, all URLs will be formed correctly.)</p>

  <p>So how does this magic work? Each site has a route datatype, and each resource pattern gets its own constructor. In our previous example, we would get something that looks like:</p>
  <pre>
data MySiteRoute = HomepageR
                 | AddEntryR
                 | EntryR EntryId

</pre>

  <p>If you want to link to the homepage, you use <code>HomepageR</code>. To link to a specific entry, you would use the <code>EntryR</code> constructor with an <code>EntryId</code> parameter. For example, to create a new entry and redirect to it, you could write:</p>
  <pre>
entryId &lt;- insert (Entry "My Entry" "Some content")
redirect RedirectTemporary (EntryR entryId)
</pre>

  <p>Hamlet, Lucius and Julius all include built-in support for these type-safe URLs. Inside a Hamlet template you can easily create a link to the add-entry page:</p>
  <pre>
&lt;a href=@{AddEntryR}&gt;Create a new entry.
</pre>

  <p>The best part? Just like Persistent entities, the compiler will keep you honest. If you change any of your routes (e.g., you want to include the year and month in your entry routes), Yesod will force you to update every single reference throughout your codebase.</p>

  <h3 id="heading_id_23">Handlers</h3>

  <p>Once you define your routes, you need to tell Yesod how you want to respond to requests. This is where <em>handler functions</em> come into play. The setup is simple: for each resource (e.g., <code>HomepageR</code>) and request method, create a function named <code>methodResourceR</code>. For our previous example, we would need four functions: <code>getHomepageR</code>, <code>getAddEntryR</code>, <code>postAddEntryR</code>, and <code>getEntryR</code>.</p>

  <p>All of the parameters collected from the route are passed in as arguments to the handler function. <code>getEntryR</code> will take a first argument of type <code>EntryId</code>, while all the other functions will take no arguments.</p>

  <p>The handler functions live in a <code>Handler</code> monad, which provides a great deal of functionality, such as redirecting, accessing sessions, and running database queries. For the last one, a typical way to start off the <code>getEntryR</code> function would be:</p>
  <pre>
getEntryR entryId = do
    entry &lt;- runDB $ get404 entryId
</pre>

  <p>This will run a database action that will get the entry associated with the given ID from the database. If there is no such entry, it will return a 404 response.</p>

  <p>Each handler function will return some value, which must be an instance of <code>HasReps</code>. This is another RESTful feature at play: instead of just returning some HTML or some JSON, you can return a value that will return either one, depending on the HTTP Accept request header. In other words, in Yesod, a resource is a specific piece of data, and it can be returned in one of many <em>representations</em>.</p>

  <h3 id="heading_id_24">Widgets</h3>

  <p>Assume you want to include a navbar on a few different pages of your site. This navbar will load up the five most recent blog posts (stored in your database), generate some HTML, and then need some CSS and Javascript to style and enhance.</p>

  <p>Without a higher-level interface to tie these components together, this could be a pain to implement. You could add the CSS to the site-wide CSS file, but that's adding extra declarations you don't always need. Likewise with the Javascript, though a bit worse: having that extra Javascript might cause problems on a page it was not intended to live on. You will also be breaking modularity by having to generate the database results from multiple handler functions.</p>

  <p>In Yesod, we have a very simple solution: widgets. A widget is a piece of code that ties together HTML, CSS and Javascript, allowing you to add content to both the head and body, and can run any arbitrary code that belongs in a handler. For example, to implement our navbar:</p>
  <pre>
-- Get last five blog posts. The "lift" says to run this code like we're in the handler.
entries &lt;- lift $ runDB $ selectList [] [LimitTo 5, Desc EntryPosted]
toWidget [hamlet|
&lt;ul .navbar&gt;
    $forall entry &lt;- entries
        &lt;li&gt;#{entryTitle entry}
|]
toWidget [lucius| .navbar { color: red } |]
toWidget [julius|alert("Some special Javascript to play with my navbar");|]
</pre>

  <p>But there is even more power at work here. When you produce a page in Yesod, the standard approach is to combine a number of widgets together into a single widget containing all your page content, and then apply <code>defaultLayout</code>. This function is defined per site, and applies the standard site layout.</p>

  <p>There are two out-of-the-box approaches to handling where the CSS and Javascript go:</p>

  <ol>
    <li>Concatenate them and place them into <code>style</code> and<code>script</code> tags, respectively, within your HTML.</li>

    <li>Place them in external files and refer to them with <code>link</code> and <code>script</code> tags, respectively.</li>
  </ol>

  <p>In addition, your Javascript can be automatically minified. Option 2 is the preferred approach, since it allows a few extra optimizations:</p>

  <ol>
    <li>The files are created with names based on a hash of the contents. This means you can place cached values far in the future without worries of users receiving stale content.</li>

    <li>Your Javascript can be asynchronously loaded.</li>
  </ol>

  <p>The second point requires a bit of elaboration. Widgets not only contain raw Javascript, they also contain a list of Javascript dependencies. For example, many sites will refer to the jQuery library and then add some Javascript that uses it. Yesod is able to automatically turn all of that into an asynchronous load via <code>yepnope.js</code>.</p>

  <p>In other words, widgets allow you to create modular, composable code that will result in incredibly efficient serving of your static resources.</p>

  <h3 id="heading_id_25">Subsites</h3>

  <p>Many websites share common areas of functionality. Perhaps the two most common examples of this are serving static files and authentication. In Yesod, you can easily drop in this code using a <em>subsite</em>. All you need to do is add an extra line to your routes. For example, to add the static subsite, you would write:</p>
  <pre>
/static StaticR Static getStatic
</pre>

  <p>The first argument tells where in the site the subsite starts. The static subsite is usually used at <code>/static</code>, but you could use whatever you want. <code>StaticR</code> is the name of the route; this is also entirely up to you, but convention is to use <code>StaticR</code>. <code>Static</code> is the name of the static subsite; this is one you do not have control over. <code>getStatic</code> is a function that returns the settings for the static site, such as where the static files are located.</p>

  <p>Like all of your handlers, the subsite handlers also have access to the <code>defaultLayout</code> function. This means that a well-designed subsite will automatically use your site skin without any extra intervention on your part.</p>

  <h2 id="heading_id_26">22.6. Lessons Learned</h2>

  <p>Yesod has been a very rewarding project to work on. It has given me an opportunity to work on a large system with a diverse group of developers. One of the things that has truly shocked me is how different the end product has become from what I had originally intended. I started off Yesod by creating a list of goals. Very few of the main features we currently tout in Yesod are in that list, and a good portion of that list is no longer something I plan to implement. The first lesson is:</p>

  <p><em>You will have a better idea of the system you need after you start working on it. Do not tie yourself down to your initial ideas.</em></p>

  <p>As this was my first major piece of Haskell code, I learned a lot about the language during Yesod's development. I'm sure others can relate to the feeling of "How did I ever write code like this?" Even though that initial code was not of the same caliber as the code we have in Yesod at this point, it was solid enough to kick-start the project. The second lesson is:</p>

  <p><em>Don't be deterred by supposed lack of mastery of the tools at hand. Write the best code you can, and keep improving it.</em></p>

  <p>One of the most difficult steps in Yesod's development was moving from a single-person team—me—to collaborating with others. It started off simply, with merging pull requests on GitHub, and eventually moved to having a number of core maintainers. I had established some of my own development patterns, which were nowhere explained or documented. As a result, contributors found it difficult to pull my latest unreleased changes and play around with them. This hindered others both when contributing and testing.</p>

  <p>When Greg Weber came aboard as another lead on Yesod, he put in place a lot of the coding standards that were sorely lacking. To compound the problems, there were some inherent difficulties playing with the Haskell development toolchain; specifically in dealing with Yesod's large number of packages. One of the goals of the entire Yesod team has since been to create standard scripts and tools to automate building. Many of these tools are making their way back into the general Haskell community. The final lesson is:</p>

  <p><em>Consider early on how to make your project approachable for others.</em></p><!-- Localized -->
</body>
</html>
